Road lane boundary detection system and road lane boundary detecting method

ABSTRACT

A road lane boundary detection system includes a detection region setting unit that sets a certain region in a road image, as a target detection region to be searched for detection of a road lane boundary, and a detecting unit that processes image data in the target detection region set by the detection region setting unit, so as to detect the road lane boundary. The detection region setting unit sets a first detection region as the target detection region if no road lane boundary is detected, and sets a second detection region as the target detection region if the road lane boundary is detected, such that the first and second detection regions are different in size from each other.

INCORPORATION BY REFERENCE

The disclosure of Japanese Patent Application No. 2008-138197 filed onMay 27, 2008 including the specification, drawings and abstract isincorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to system and method for detecting road laneboundaries, and more particularly to such system and method fordetecting road lane boundaries, based on image data representing acaptured image of a road ahead of the vehicle.

2. Description of the Related Art

A road lane boundary detection system for a vehicle is known which isoperable to detect road lane boundaries, such as white lines, which aredrawn on a road and define a lane in which the vehicle runs. This systemis configured to detect road lane boundaries by processing an image(image data) captured by a camera installed on the own vehicle fortaking pictures of a region ahead of the vehicle. For example, thesystem extracts an image of a certain specific size to be compared, fromthe captured image, and determines whether the extracted image to becompared matches any of registered images relating to particularobjects, using image processing, such as digitization, edge detectionand filtering, of the image. If it is determined that the extractedimage matches a registered image relating to a particular object, it isrecognized that the particular object is present in the captured image.In this manner, white lines, or the like, are detected. The road laneboundary detection system of this type is required to detect whitelines, or the like, on the road ahead of the vehicle, from moment tomoment at very short time intervals of 1/10 sec. to 1/100 sec. However,an enormous amount of computations are needed to carry out a detectingprocess for detecting white lines, or the like, with respect to all ofthe pixels of the captured image. Accordingly, various approximations orsimplifications are performed in each step of the process leading todetection of white lines, or the like.

For example, an image recognition system (as disclosed in, for example,Japanese Patent Application Publication No. 2007-235414(JP-A-2007-235414)) is configured to limit a range to be searched for acertain object or objects, in an image captured from a mobile unit, soas to reduce the load on the system for image recognition processing.With this technology, a range within which imaging is possible isdivided into a range delimited by a boundary to which the vehicle isable to reach after a lapse of a specified time, and a range beyond theboundary, into which the vehicle is not able to reach within thespecified time. Then, the system limits a range on which the imagerecognition process is performed, to the range that can be reached bythe vehicle, so as to reduce the processing load on the system forprocessing, such as extraction of an image to be compared with theregistered image(s), and comparison between the image to be compared andthe registered image(s).

Also, a system (as disclosed in, for example, Japanese PatentApplication Publication No. 2007-72512 (JP-A-2007-72512)) is configuredto set a close region that is relatively close to the vehicle and aremote region that is farther than the close region, in an imagecaptured from a mobile unit, for use in detection of road laneboundaries, such as solid lines or broken lines. With this technology,when the system reads image data, it divides a region to be processed inthe image, into the close region that is relatively close to thevehicle, and the remote region that is farther than the close region,and set these regions for use in subsequent processing. Also, the systemhas a solid line detection mode for detecting solid lines, and anintermittent line detection mode for detecting broken lines. In thesolid line detection mode, the system sets twenty detection linesparallel to the X axis so as to equally divide the distance on the Yaxis in the region into twenty sections, with respect to each of theclose region and the remote region. In the intermittent line detectionmode, on the other hand, the system sets forty detection lines parallelto the X axis so as to equally divide the distance on the Y axis in theregion into forty sections, with respect to the close region, forexample. Here, the detection lines set on the image provide equallyspaced lines in real space. FIG. 16 is a view showing the correspondingpositions of the detection lines in the real space, which indicatesdetection lines on the road as viewed from above. Since the detectionlines are arranged so as to be spaced at equal intervals in the realspace, as shown in FIG. 16, the detection lines provide unequally spacedlines on the image, as shown in FIG. 17. Once the system sets thedetection lines in this manner, it differentiates the brightness(density) of pixel in the X-axis direction, with respect to each pixelon each detection line, and extracts or picks up coordinates of pointsat which the absolute value of the differential exceeds a thresholdvalue A, as feature points, based on which road lane boundaries aredetected.

However, the systems as disclosed in the above-identified patentpublications suffer from some problems as follows. First, so-calledBotts dots (dots each having a diameter of about 10 cm) are known as onetype of road lane boundaries or markings provided on the road in placeof white lines. Unlike the white lines, the Botts dots areintermittently arranged or spaced at given intervals on the road. On theroad on which each road lane boundary consists of these intermittentmarkings, it would be easy to assume or recognize a single road laneboundary from these intermittent markings via human eyes. However, inthe case where a system, like the systems as described above,mechanically recognizes road lane boundaries in the form of imaginary orsubstantial lines, through image processing on an image that is capturedby a camera and that includes the intermittent markings, the road laneboundaries may not be appropriately detected. Even if the region to beprocessed on the captured image is limited as in the system ofJP-A-2007-235414, and a detecting process is performed using spaceddetection lines by which the distance on the Y axis in the region isequally divided into twenty or forty sections, as in the system ofJP-A-2007-72512, feature points, or the like, associated with Botts dotsthat are present between the detection lines cannot be acquired, due tothe fact that the Botts dots are intermittently arranged, and accuratedetection cannot be achieved.

It may be proposed to subject all lines (all pixels) of the capturedimage to the processing for detection, without extracting or selectingdetection lines from all lines. In this case, however, an enormousamount of computations are needed, and the processing load increases,thus making it difficult to quickly acquire necessary information. Itmay also be proposed to divide the region to be processed in the image,into a close region that is relatively close to the vehicle and a remoteregion that is farther than the close region, and limit the object to beprocessed to the close region so that all lines of the close region aresearched for detection, thereby to reduce the processing load. Althoughthis method is useful in a detecting process performed, for example,when the vehicle starts running, or for the purpose of “finding” Bottsdots, etc., it is necessary to continue searching for Botts dots on theroad (i.e., tracking Botts dots) after the Botts dots are initiallyfound, so as to recognize the Botts dots as road lane boundaries. Inthis case, since the object to be processed is limited to the closeregion, the detecting process is only performed on, for example, aregion up to 10 m ahead of the vehicle, resulting in a problem with theaccuracy with which Botts dots are detected during running of thevehicle, for example.

SUMMARY OF THE INVENTION

The present invention provides system and method for detecting road laneboundaries that are able to accurately recognize lane boundaries, suchas Botts dots or white lines, without increasing the processing load.

According to a first aspect of the invention, there is provided a roadlane boundary detection system for detecting a road lane boundaryprovided on a road, from a road image that is a captured image of a roadahead of a vehicle, which includes a detection region setting unit thatsets a certain region in the road image, as a target detection region tobe searched for detection of the road lane boundary, and a detectingunit that processes image data in the target detection region set by thedetection region setting unit, so as to detect the road lane boundary.In the road lane boundary detection system, the detection region settingunit sets a first detection region as the target detection region if noroad lane boundary is detected, and sets a second detection region asthe target detection region if the road lane boundary is detected, suchthat the first detection region and the second detection region aredifferent in size from each other.

According to the first aspect of the invention, it is possible to detectroad lane boundaries with high accuracy, while suppressing or avoidingan increase in the processing load on a processor of the system.

In the system of the first aspect of the invention, the detection regionsetting unit may set the second detection region such that the width ofthe second detection region is smaller than that of the first detectionregion.

With the above arrangement, the same advantageous effects as those ofthe first aspect of the invention can be obtained.

In the system as described above, the detection region setting unit mayinclude: an initial detection region setting unit that sets, in the roadimage, an initial detection region of a certain size which is positionedat a first distance from the vehicle in real space and has a firstdimension as measured in a width direction of the road, as the targetdetection region, a close detection region setting unit that sets aclose detection region that is positioned at a distance substantiallyequal to the first distance from the vehicle in real space, and has asecond dimension as measured in the road width direction, which issmaller than the first dimension of the initial detection region, as thetarget detection region, and a remote detection region setting unit thatsets a remote detection region that is positioned at a second distancethat is larger than the first distance from the vehicle in real space,and has a third dimension as measured in the road width direction, whichis smaller than the first dimension of the initial detection region, asthe target detection region. In this system, the detecting unit mayprocess image data in the initial detection region if no road laneboundary is detected, so as to detect a road lane boundary, and mayprocess image data in the close detection region and the remotedetection region if the road lane boundary is detected, so as to detectthe road lane boundary.

With the above arrangement, when no road lane boundary is detected, arelatively wide range of only the close region of the vehicle, in theroad image, is set as a detection region to be searched for detection ofa road lane boundary. Once any road lane boundary is detected, thedetection range is narrowed, and the narrowed range of the close region,and the remote region, are searched for the road lane boundary. It isthus possible to detect road lane boundaries with high accuracy, withoutincreasing the processing load.

In the system as described above, the close detection region settingunit may set a position of the close detection region as viewed in theroad width direction in the road image, based on a position of the roadlane boundary detected by the detecting unit using the initial detectionregion, and the remote detection region setting unit may set a positionof the remote detection region as viewed in the road width direction inthe road image, based on the position of the road lane boundary detectedby the detecting unit using the initial detection region.

In the system as described just above, the close detection regionsetting unit my set the position of the close detection region in theroad image, such that the road lane boundary detected by the detectingunit using the initial detection region is located at a center of theclose detection region as viewed in the road width direction, and theremote detection region setting unit may set the position of the remotedetection region in the road image, such that the road lane boundarydetected by the detecting unit using the initial detection region islocated at a center of the remote detection region as viewed in the roadwidth direction.

With the arrangements as described above, the positions of the closedetection region and remote detection region are set based on theposition of the initially detected road lane boundary in the road image.It is thus possible to detect road lane boundaries with high accuracy,without increasing the processing load.

In the system of the first aspect of the invention, the target detectionregion may comprise a pair of regions that are separated from each otherin the road width direction, and each of the regions may have a widththat contains each of left and right lane boundaries located to the leftand right of the vehicle.

With the above arrangement, the processing load arising in the detectionprocess can be further reduced.

In the system of the first aspect of the invention, the road laneboundary may be in the form of Botts dots.

In the above case, the system is able to detect Botts dots that areintermittently arranged on the road, with high accuracy, whilesuppressing or avoiding an increase in the processing load.

According to a second aspect of the invention, there is provided a roadlane boundary detecting method for detecting a road lane boundaryprovided on a road, from a road image that is a captured image of a roadahead of a vehicle, which includes: setting a certain region in the roadimage, as a target detection region to be searched for detection of theroad lane boundary, and processing image data in the target detectionregion so as to detect the road lane boundary. In this method, a firstdetection region is set as the target detection region for use in acondition where no road lane boundary has been detected, and a seconddetection region is set as the target detection region for use in acondition where the road lane boundary has been detected, such that thefirst detection region and the second detection region are different insize from each other.

According to the second aspect of the invention, the same advantageouseffects as those of the first aspect of the invention can be obtained.

BRIEF DESCRIPTION OF THE DRAWINGS

The features, advantages, and technical and industrial significance ofthis invention will be described in the following detailed descriptionof example embodiments of the invention with reference to theaccompanying drawings, in which like numerals denote like elements, andwherein:

FIG. 1 is a functional block diagram of a road lane boundary detectionsystem according to one embodiment of the invention;

FIG. 2 is a view schematically showing a camera image;

FIG. 3 is a view schematically showing a camera image;

FIG. 4 is a view showing one example of initial detection areas;

FIG. 5 is a view showing the corresponding positions of the initialdetection areas in real space;

FIG. 6 is a view showing one example of areas to be searched in atracking process;

FIG. 7 is a view showing the corresponding positions of the areas to besearched in the tracking process, in real space;

FIG. 8 is a view useful for explaining setting of an area to besearched;

FIG. 9 is a view useful for explaining setting of areas to be searched;

FIG. 10 is a view useful for explaining setting of areas to be searched;

FIG. 11 is an illustration showing a memory map of a RAM shown in FIG.1;

FIG. 12 is a flowchart illustrating the flow of a detecting processperformed in the detection system;

FIG. 13 is a view useful for explaining feature points of a white line;

FIG. 14 is a flowchart illustrating details of a Botts-dots detectionprocess indicated at step S9 in FIG. 12;

FIG. 15 is a flowchart illustrating the flow of a detecting processaccording to a second embodiment of the invention;

FIG. 16 is a view showing detection lines of the related art in realspace; and

FIG. 17 is a view showing the detection lines on an image.

DETAILED DESCRIPTION OF EMBODIMENTS

An exemplary embodiment of the invention will be described withreference to the drawings. It is to be understood that the invention isnot limited to this embodiment.

FIG. 1 is a functional block diagram showing an example of theconfiguration of a road lane boundary detection system 10 (which will besimply called “detection system”) according to the present invention.The detection system 10 consists essentially of an ECU (ElectronicControl Unit) 11, NVRAM (Non-Volatile Random Access Memory) 12, RAM(Random Access Memory) 13, and a camera 15.

The ECU 11 performs certain computations, based on a signal from thecamera 15 and information stored in the NVRAM 12 and the RAM 13, so asto perform processing as will be described later.

The NVRAM 12 is a non-volatile rewritable storage medium, and storesprograms, or the like, for executing procedures as will be describedlater. The ECU 11 retrieves these programs from the NVRAM 12 and expandsthem in the RAM 13, so that the detecting system 10 performs a detectingprocess according to this embodiment of the invention.

The RAM 13 is a volatile, rewritable storage medium, and is used as aworkspace of the ECU 11. Also, the RAM 13 temporarily stores datareceived from the camera 15.

The camera 15 is a vehicle-mounted imaging means, and may be, forexample, a CCD (Charge-Coupled Device) camera or CMOS (ComplementaryMetal-Oxide Semiconductor) camera installed at a location (e.g., on afront grille or in the vicinity of the room mirror in the vehiclecompartment) which is suitable for capturing a basic image as viewed inthe travelling direction of the vehicle or an image of the environmentsurrounding the vehicle. The imaging means may also be a video camerafor taking moving video pictures, or a still camera for taking stillpictures, or an infrared camera capable of taking pictures in thenighttime.

The NVRAM 12 also stores information concerning the camera 15, includingthe angle of the field of view indicating the range over which picturescan be taken, the angle of installation, shutter speed, the total numberof pixels, f number or stop number, shooting distance, shooting delaytime as a period of time from a certain picture-taking to the next oneor until image processing for the next image frame becomes possible.

Referring next to FIG. 2 through FIG. 10, a road lane boundary detectionprocess contemplated in this embodiment will be generally described. Inthe lane boundary detection process of this embodiment, an image of aroad ahead of the vehicle, which was picked up by the camera 15, iscaptured, and suitable image processing (such as line scanning) as willbe described later is performed on the captured image, for detection ofroad lane boundaries. The road lane boundary detection process isstarted, for example, at the time when the ignition switch is turned on.The content of this process is roughly divided into a process (whichwill be called “initial finding process”) for finding road laneboundaries, such as white lines or Botts dots, and a process (which willbe called “tracking process”) for tracking the road lane boundaries thusfound. At first (i.e., before the vehicle starts running), the initialfinding process is executed for finding road lane boundaries.Subsequently, the tracking process is executed for continuing detecting(recognizing) the road lane boundaries during vehicle running, based onthe position, shape, etc. of each of the lane boundaries found in theinitial finding process. Then, various functions, such as a lane keepingassist function, are performed, based on the lane boundaries detected inthis manner. In the following explanation, white lines and Botts dotsare taken as examples of the road lane defining lines. The white linesmentioned herein include “solid lines” that are continuously drawn onthe road, and “broken lines” that consist of short lines that aresuccessively arranged in line at substantially equal intervals.

FIG. 2 and FIG. 3 schematically show camera images (road images)captured by the camera 15. FIG. 2 is a camera image in which white linesare illustrated as an example of road lane boundaries, and FIG. 3 is acamera image in which Botts dots are illustrated as an example of roadlane boundaries.

In this embodiment, a process for detecting white lines is initiallyexecuted as the above-mentioned initial finding process. Here, anymethod for detecting white lines may be employed. For example, thewhite-line detection process is performed by setting suitably extractedor selected detection lines on a camera image, and performing linescanning on the detection lines so as to calculate feature points of,for example, the brightness. Because white lines are not discontinuouslyarranged (i.e., extend continuously), detection of these lines ispossible even if the extracted detection lines are used. If any whiteline is detected as a result of the above process, the white linecontinues to be detected in the subsequent tracking process. With thethus detected white lines being regarded as road lane boundaries,various functions, such as a lane keeping assist function, are executed.

If no white line is detected in the initial finding process, a processfor detecting Botts dots (in other words, initial finding of Botts dots)is then executed. When the system attempts to detect Botts dots in acamera image, using line scanning, the extraction of the detection lines(i.e., reduction of the number of detection lines) as described aboveshould not be conducted since the Botts dots are discontinuouslyarranged on the road. However, if the detecting process is performedwith respect to all of the pixels (all lines) of the camera image, theprocessing load will increase. In this embodiment, therefore, targetdetection areas to be searched for detection of Botts dots are set asshown in FIG. 4, in order to reduce the processing load. FIG. 4schematically shows target detection areas 101 a, 101 b to be searchedfor detection of Botts dots during the initial finding process. Thesedetection areas 101 a, 101 b will be called “initial detection areas”.In the following explanation, the areas 101 a, 101 b may be simplycalled “initial detection area 101” as a generic term of these twoareas. In FIG. 4, the X axis represents the width direction of the road,and the Y axis represents the longitudinal direction of the road. Asshown in FIG. 4, the initial detection areas 101 a, 101 b are set asleft and right areas in a lower section of the camera image (namely, onone side closer to the vehicle). FIG. 5 is a bird's eye view of FIG. 4,showing the initial detection areas 101 a and 101 b of FIG. 4 as viewedfrom above in real space or on real coordinates. As shown in FIG. 5, theinitial detection areas 101 a and 101 b are set such that Botts dots areplaced at the center on the X axis of each of the areas. A detectingprocess is performed on all lines contained in the initial detectionareas 101 a and 101 b, for detection of Botts dots. In this embodiment,top-hat conversion of morphology operations is used for detection ofBotts dots, as will be described in detail later.

If Botts dots are detected, the system continues detecting or monitorsBotts dots in the subsequent tracking process. In the tracking process,areas as shown in FIG. 6 are set as target detection areas. FIG. 6schematically shows target detection areas (which will be called“tracking target detection areas”) to be searched during the trackingprocess for detection of Botts dots. FIG. 7 is a bird's eye view of FIG.6, showing the tracking target detection areas of FIG. 6 as viewed fromabove in real space. As shown in FIG. 6, the tracking target detectionareas include two areas 102 a and 102 b as left and right areas in alower section (closer to the vehicle) of the camera image, and two areas103 a and 103 b as left and right areas located above the areas 102 a,102 b, respectively. The areas 102 a and 102 b will be called “closedetection areas”. In the following description, the areas 102 a, 102 bmay be simply called “close detection area 102” as a generic term ofthese two areas. The areas 103 a and 103 b will be called “remotedetection areas”. In the following description, the areas 103 a, 103 bmay be simply called “remote detection area 103” as a generic term ofthese two areas. On the camera image as shown in FIG. 6, the width orlateral dimension of the close detection areas 102 a and 102 b issomewhat narrower or smaller than that of the initial detection areas101 a and 101 b. Also, the width or lateral dimension of the remotedetection areas 103 a and 103 b is further narrower than that of theclose detection areas 102 a and 102 b. In the real space as shown inFIG. 7, the close detection area 102 and the remote detection area 103are depicted as regions having the same size or area. Each of theseregions is set as a range within which a road lane boundary is likely tobe present on the road (the manner of setting the range will bedescribed later). Thus, even where the regions have the same size in thereal space, a remote one of the regions is indicated as a smaller regionon the camera image. Therefore, on the camera image as shown in FIG. 6,the width of the remote detection areas 103 a and 103 b as measured inthe camera image is further narrower than that of the close detectionareas 102 a and 102 b. Namely, the remote detection areas 103 a and 103b are set as areas having further narrowed detection ranges. As thedetection range becomes narrower, the processing load associated withthe detecting process as described above is reduced.

The position and size of each detection area including the range withinwhich a road lane boundary is likely to present on the road arecalculated in advance, and data indicative of the results of calculationis stored in advance in a memory, or the like, as set data of thedetection area. In operation, the set data is read from the memory andused in a process as will be described later. For calculation of theposition and size of each of the detection areas, the vehicle is placedon a road of, for example, a test course or test track, on which whitelines are drawn, and the position and size of the detection area arecalculated in the manner as will be described below while the vehicle isactually running with white lines being detected, for example. Referringto FIG. 8 to FIG. 10, the manner of setting or calculating the positionand size of each of the initial detection areas 101, close detectionareas 102 and remote detection areas 103 will be explained.

Initially, as shown in FIG. 8, a vertical dimension, or a range asmeasured in the vertical direction, of an area to be searched fordetection is set on a camera image. In the example of FIG. 8, a rangedelimited by two horizontal lines (bounds) ahead of the vehicle, whichare spaced 7 m and 27 m, respectively, from the vehicle, in the realspace is set as one example of target detection area on the cameraimage. One of the bounds of the range is set at 7 m ahead of thevehicle, in view of interruption by a bonnet (or hood) of a passengercar. The other bound is set at 27 mm ahead of the vehicle, in view ofthe resolution of the image, the range (30 m) of illumination in thenighttime, and so forth.

Next, the area of which the vertical dimension has been set is furtherdivided into a close region that is relatively close to the vehicle, anda remote region that is farther than the close region, as shown in FIG.9.

Furthermore, the width or lateral dimension is calculated assuming asituation where the vehicle makes a lane change. More specifically,supposing that the vehicle is located at the center of a lane, as shownin FIG. 10, the width that contains road lane boundaries (white lines inFIG. 10) at the outer sides of the left and right lanes is calculatedand set. In addition, the position of the detection area with respect tothe X axis is adjusted such that the FOE (Focus of Expansion) lies on acenter line (vertical line in FIG. 10) located at the center on the Xaxis.

Next, the set content of the initial detection areas 101 a and 101 b iscalculated from the regions set in the manner as described above. Theinitial detection areas 101 a and 101 b are set using only the closeregion. Initially, the close region is divided into left and rightregions that are opposed to each other with respect to the center linethat passes the FOE. Then, the width of each of the left and rightregions is calculated. To calculate the width of each region, a processfor detecting a road lane boundary (typically, a white line) isperformed with reference to the position of the own vehicle at the timeof a lane change as shown in FIG. 10, and the width is calculated andset in view of the R value (e.g., 200R) of a curve, with reference tothe position of the detected road lane boundary. Namely, the width iscalculated and set such that the lane (or lanes) whose image is capturedby the camera 15 does not come out of the target detection area whilethe vehicle is running along a curve. At this time, the angle of viewand resolution of the camera are also taken into consideration. As aresult, the initial detection areas 101 a and 101 b as shown in FIG. 4are calculated.

Next, the set content of the close detection areas 102 a and 102 b usedin the tracking process is calculated. The respective positions of theclose detection areas 102 a and 102 b on the X axis are determined withreference to the positions of the road lane boundaries found uponcalculation and setting of the set content of the initial detectionareas 101. Namely, each of the close detection areas 102 a and 102 b ispositioned so that the corresponding road lane boundary lies at thecenter of the area 102 a, 102 b on the X-axis. On the other hand, thewidth or lateral dimension is calculated in view of the followingpoints.

(1) Initially, the distance from the road lane boundary corresponding toeach of the detection areas 102 to the road shoulder is taken intoconsideration, so that the road shoulder is prevented from beingerroneously recognized as a road lane boundary. Therefore, the width ofthe close detection areas 102 a and 102 b is calculated and set to benarrower or smaller than that of the initial detection areas 101 a and101 b.(2) In addition, the behavior of the vehicle is taken intoconsideration. More specifically, the following two points are takeninto consideration. First, a shift or variation in the display positionof road lane boundaries from one frame of camera image to the nextframe, which occurs when the vehicle makes a lateral movement, is takeninto consideration. Suppose that one frame lasts 1/10 second (namely,camera images are picked up at the intervals of 1/10 second), and thevehicle is running with the steering wheel turned to the right. In thiscase, with respect to the positions of road lane boundaries, e.g., whitelines, in a certain frame of camera image, the positions of the whitelines in the next frame of camera image are slightly shifted to theleft. The width of the close detection areas is calculated so that theroad lane boundaries are contained in the close detection areas evenwhen the vehicle makes a lateral movement, as in the above case.Secondly, the yaw angle of the vehicle sensed when the vehicle isoriented in a slanting direction with respect to the road laneboundaries is taken into consideration, assuming the case where thevehicle makes a lane change, for example. Upon a lane change, thelongitudinal axis or running direction of the vehicle becomes inclinedby one to two degrees with respect to the road lane boundaries, such aswhite lines. Therefore, the white lines that appear in the camera imageare also slightly inclined. In the case of a vehicle on which aso-called lane keeping assist system is installed, for example, when thevehicle is about to depart from the lane on which the vehicle iscurrently running, appropriate torque is applied to an electric powersteering, so as to assist the vehicle in keeping running on the currentlane. The width of the close detection areas is calculated so that thewhite lines are contained in the close detection areas, in view of theyaw angle (the vehicle is temporarily oriented in a slanting directionwith respect to the white lines) due to the torque produced in the abovesituation. The width of the close detection areas is calculated in viewof the above-described points, and stored in a memory as set data.

Next, the remote detection areas 103 as shown in FIG. 6 are calculated.More specifically, the same points as those considered in calculation ofthe close detection areas 102 are examined with regard to theabove-mentioned remote region, and the remote detection areas 103 arecalculated and set in view of these points. It is, however, to be notedthat, on the camera image, the road lane boundaries in the remote regionappear in smaller size (or thinner) than those in the close region. Thewidth of the remote detection areas 103, which is calculated based onthe road lane boundaries, is set to a value that is further smaller ornarrower than that of the close detection areas 102.

The target detection area to be searched for detection of Botts dots islimited to the close region in the initial finding stage, and includesor covers the remote region as well as the close region in the trackingstage, for the following reason. In the initial finding stage (where thepresence of Botts dots has not yet been recognized), it is only requiredto grab the position of Botts dots, and there is a little necessity toinclude the remote region in the detection area to be searched. In thetracking stage, typically during running of the vehicle, it is requiredto detect Botts dots in a remote region, and keep recognizing the Bottsdots as road lane boundaries. Thus, in the initial finding stage, onlythe close region is searched for detection of Botts dots, so as to graspthe position of the Botts dots without increasing the processing load.Since the purpose of the initial finding process is to initially find ordetect Botts dots, the search range (the width of the area to besearched for detection) is set to be larger than that of the trackingprocess. Once the position of the Botts dots is found in the initialfinding stage, the X-axis position of the Botts dots in the camera imageis presumed not to change largely during subsequent running of thevehicle. Therefore, even if the width of the detection area is reducedwith the thus found X-axis position set as the center of the area on theX axis, subsequent detection or monitoring of the Botts dots may besufficiently accomplished. Thus, the width of the detection area in thetracking stage is set to be further smaller than that in the initialfinding stage, so as to reduce the processing load. In the remoteregion, Botts dots appear in further reduced size on the camera image.Therefore, detection of Botts dots is possible even if the width of theremote detection areas 103 is set to be further smaller than that of theclose detection areas 102, as shown in FIG. 6. While the processing loadis reduced by reducing the width of the close detection areas 102, theprocessing power that has increased with the reduction in the processingload is utilized in the detecting process performed on the remotedetection areas 103. As a result, the Botts-dots detecting process canbe performed on both the close region and the remote region, withoutincreasing the overall processing load.

In this embodiment, the Botts-dots detecting process is performed withthe size of the detection area varied between the initial finding stageand the tracking stage, as described above. Thus, while the Botts-dotsdetection process generally requires large processing load associatedwith image processing, the above arrangement makes it possible toaccomplish detection of Botts dots with high accuracy while keeping theoverall processing load at a sufficiently low level.

In the following, various kinds of data and programs used in thisembodiment will be described in detail. Initially, various kinds of dataused in this embodiment will be described. FIG. 11 is an illustrationshowing a memory map of the RAM 13 as shown in FIG. 1. In FIG. 11, theRAM 13 includes a program storage region 131 and a data storage region133. Data stored in the program storage region 131 and the data storageregion 133, which is stored in advance in the NVRAM 12, is copied fromthe NVRAM 12 to the RAM 13.

A detection program 132 to be executed by the ECU 11 is stored in theprogram storage region 131.

Various kinds of data, including initial detection area data 134, closedetection area data 135, remote detection area data 136, processing mode137, and detection mode 138, are stored in the data storage region 133.

The initial detection area data 134 indicates the position and size ofeach of the above-described initial detection areas 101 a and 101 b onthe camera image.

The close detection area data 135 indicates the size of theabove-described close detection areas 102 a and 102 b on the cameraimage. The remote detection area data 136 indicates the size of theabove-described remote detection areas 103 a and 103 b on the cameraimage.

The date of the processing mode 137, which is a variable used inroutines that will be described later, is a variable indicating which ofthe initial finding process and the tracking process as described aboveis to be executed.

The date of the detection mode 138, which is a variable used in routinesthat will be described later, is a variable used for determining whether“white lines” or “Botts dots” are to be detected.

Next the control flow of a detecting process performed in the detectingsystem 10 will be described with reference to FIG. 12 through FIG. 14.FIG. 12 is a flowchart illustrating the flow of the detecting processperformed in the detecting system 10. When the ignition switch of thevehicle is turned on (i.e., when the power is turned on), the ECU 11initializes each unit of the RAM 13, etc. Then, the detection program,etc. stored in the NVRAM 12 is read into the RAM 13 and execution of thedetection program is started. Namely, when the ignition switch is turnedon, a control routine as shown in the flowchart of FIG. 12 is started.In this embodiment, various processes performed in situations other thanthat of the detecting process are not directly related to the presentinvention, and therefore will not be explained herein. The processingloop of step S1 through step S11 as shown in FIG. 12 is repeatedlyexecuted for each frame (i.e., at the intervals of 1/10 second in thisembodiment).

In FIG. 12, the ECU 11 initially sets the processing mode 137 to“INITIAL DETECTION” (step S1). Subsequently, the ECU 11 acquires acamera image (more precisely, camera image data) from the camera 15(step S2), and executes a process for detecting a white line from theimage (step S3). More specifically, a retrieval region of a certain sizeis set in the acquired camera image. Then, scanning is conducted in thehorizontal direction for each line in the retrieval region, and featurepoints of a white line (i.e., edge points: an edge that indicates theleft-side boundary of the white line will be called “rise edge”, and anedge that indicates the right-side boundary of the white line will becalled “fall edge”) are detected. Since a white line generally looksbrighter, i.e., has a higher degree of brightness, than the roadsurface, those of pixels on each detection (or scanning) line, at whichthe brightness changes largely, are extracted as the feature points. Theamount of change in the brightness is calculated using, for example, adifferentiating filter. The feature points calculated in this mannerrepresent boundary points (corresponding to the right end and left endof the white line) between the white line and the road. FIG. 13 is aview schematically showing feature points of a white line. In FIG. 13,the upper graph indicates the brightness measured along a scanning lineon a camera image (detection area), and the lower graph indicates theamount of change in the brightness calculated by using thedifferentiating filter. When a white line is detected, as shown in FIG.13, a pair of feature points 201 and 202 are present (the feature point201 represents a rise edge, and the feature point 202 represents a falledge). To the contrary, feature points that fail to provide a pair aredetermined as unnecessary feature points. Then, the unnecessary featurepoints are removed, and only the paired feature points, i.e., namely,the feature points that possibly provide a white line remain, thuspermitting detection of a white line. Then, it is finally determinedwhether the feature points indicate a white line, based on the distancebetween the paired feature points, the positional relationship withfeature points on other lines, and so forth. It is to be understood thatthe above-described method is a mere example but not a limiting one, andthat any method may be used provided that a white line can be detectedfrom a camera image.

Next, the ECU 11 determines whether a white line was detected within apredetermined time in step S3 (step S4). If it is determined that awhite line was detected within the predetermined time in step S3 (i.e.,if an affirmative decision (YES) is obtained in step S4), the ECU 11sets “WHITE LINE” as the detection mode 138 (step S5). If no white linewas detected within the predetermined time in step S3 (i.e., if anegative decision (NO) is obtained in step S4), on the other hand, theECU 11 sets “BOTTS DOTS” as the detection mode 138 (step S6).

Next, the ECU 11 determines whether the detection mode 138 is “WHITELINE” (step S7). If it is determined that the detection mode 138 is“WHITE LINE”, a process for detecting a white line from a camera imageis carried out (step S8). The content of this process is similar to thatof step S3 as described above, and therefore will not be explained. Ifthe detection mode 138 is not “WHITE LINE”, on the other hand, thedetection mode 138 is “BOTTS DOTS”, and the ECU 11 performs a Botts-dotsdetecting process (step S9).

FIG. 14 is a flowchart showing details of the Botts-dots detectingprocess as indicated in step S9. In FIG. 14, the ECU 11 initiallydetermines whether the processing mode 137 is “INITIAL DETECTION” (stepS21). If it is determined that the processing mode 137 is “INITIALDETECTION” (i.e., if an affirmative decision (YES) is obtained in stepS21), an initial finding process for detecting Botts dots is performed.In this process, the ECU 11 sets the initial detection areas 101 (seeFIG. 4) as described above (step S22). More specifically, the ECU 11reads initial detection area data 134 from the RAM 13. Then, the ECU 11sets the initial detection areas 101 as described above in the cameraimage acquired in step S2, based on the initial detection area data 134.

Subsequently, the ECU 11 performs an operation to detect Botts dots,with respect to the above-mentioned target initial detection areas (stepS23). In this embodiment, top-hat conversion of morphology operations isused for detection of Botts dots. Initially, an image that has beensubjected to “opening” processing is subtracted from the camera image.More specifically, the camera image is set as an image to be processed,and elements corresponding to Botts dots are set as “structuralelements”. The size of the structural elements is set to 10 cm, which isthe size of Botts dots. Then, erosion (contraction) processing isperformed. As a result, pixels having relatively low brightness erodepixels having relatively high brightness, and Botts dots which aresmaller in size than the structural elements and have high brightness(more precisely, those that look like Botts dots, which will be called“Botts-dots candidates”) disappear. Then, dilation processing isperformed on the image that has contracted through the erosionprocessing. In the dilation processing, pixels having relatively highbrightness erode pixels having relatively low brightness. Although theoriginal image is restored if the once-contracted image is dilated orexpanded, the Botts-dots candidates that have disappeared will not berestored. The processing up to this step is called “opening” processing.Consequently, an image (that has been subjected to the “opening”processing) is created in which the Botts-dots candidates havedisappeared from the camera image.

Next, the image that has been subjected to the “opening” processing issubtracted from the camera image (namely, the top-hat conversion isperformed), so as to create an image in which only the Botts-dotscandidates that have disappeared through the “opening” processingremain. In the image thus created, the background is completelyeliminated, and the Botts-dots candidates remain while keeping theiroriginal shapes.

Next, the ECU 11 extracts feature points of Botts dots, based on changesin the brightness of the image on which the top-hat conversion has beendone. Then, Botts dots are detected based on the feature points. Forexample, each of the feature points thus extracted is regarded as a falledge (corresponding to the feature point 202 in FIG. 13) of a white linein the white line detecting process as described above. Then, animaginary rise edge is set at a certain distance (corresponding to thewidth of a typical white line) from the feature point in question. Inthis manner, a quasi white line can be set, on which the same operationas that used in the white line detecting process as described above isperformed, so as to detect Botts dots as a road lane boundary (moreprecisely, detect Botts dots assuming that they are a quasi white line).The method as described above is a mere example, but not a limiting one,and any method may be used provided that Botts dots can be detected froma camera image.

Next, the ECU 11 determines whether Botts dots are detected within apredetermined time in step S23 (step S24). If it is determined thatBotts dots are detected within the predetermined time in step S23 (i.e.,if an affirmative decision (YES) is obtained in step S24), the ECU 11sets the processing mode to “TRACKING” (step S25), and the Botts-dotsdetecting operation is finished. If, on the other hand, no Botts dotsare detected within the predetermined time in step S23 (i.e., if anegative decision (NO) is obtained in step S24), the control returns tostep S2, and the routine is repeated.

If it is determined in step S21 that the processing mode 137 is not“INITIAL DETECTION” (i.e., if a negative decision (NO) is obtained instep S21), on the other hand, a tracking process is executed. Initially,the ECU 11 performs an operation to set the close detection areas 102(see FIG. 6) as described above (step S26). More specifically, the ECU11 reads close detection area data 135 from the RAM 13. Then, the ECU 11sets the close detection areas 102 a and 102 b as described above in theabove-mentioned camera image, based on the close detection area data135. At this time, the close detection areas 102 a and 102 b are setwith reference to the positions of the Botts dots detected in step S23.More specifically, the ECU 11 sets the close detection area 102 a suchthat the Botts dots detected in the initial detection area 101 a in stepS23 are positioned at the center of the close detection area 102 a asviewed in the X-axis direction, namely, such that the Botts dots lie onthe center line of the close detection area 102 a as viewed in theX-axis direction. Likewise, the ECU 11 sets the close detection area 102b is set such that the Botts dots detected in the initial detection area101 b are positioned at the center or on the center line of the closedetection area 102 b as viewed in the X-axis direction.

Subsequently, the ECU 11 performs an operation to set the remotedetection areas 103 (see FIG. 6) as described above (step S27). Morespecifically, the ECU 11 reads remote detection area data 136 from theRAM 13. Then, the ECU 11 sets the remote detection areas 103 a and 103 bas described above in the above-mentioned camera image, based on theremote detection area data 136. At this time, the ECU 11 sets the remotedetection area 103 a such that the center or midpoint of the bottom sideof the remote detection area 103 a as viewed in the X-axis directionlies on the center or midpoint of the top side of the close detectionarea 102 a as viewed in the X-axis direction, (namely, the X-axiscenters of the close detection area 102 a and remote detection area 103a at the boundary thereof coincide with each other). Similarly, the ECU11 sets the remote detection area 103 b so that the X-axis centers ofthe remote detection area 103 b and the close detection area 102 bcoincide with each other (i.e., the center lines of these areas 103 b,102 b as viewed in the X-axis direction align with each other). Namely,the remote detection areas 103 a, 103 b are set so that the left andright arrays of Botts dots detected in step S23 are positioned at thecenters of the respective remote detection areas 103 a, 103 b as viewedin the X-axis direction.

Next, the ECU 11 performs an operation to detect Botts dots, withrespect to the close detection areas 102 and the remote detection areas103 (step S28). The Botts-dots detecting operation is similar to theoperation of step S23 as described above. Therefore, the detectingoperation of step S28 will not be described in detail. After executionof step S28, the Botts-dots detecting process of FIG. 14 is finished.

Returning to FIG. 12, if the process of step S8 or step S9 is finished,the ECU 11 performs a certain operation or function (step S10), based onthe detected road lane boundaries (i.e., white lines or Botts dots). Forexample, control of the steering wheel (such as so-called lane keepingassist) is performed as needed. Then, after the ECU 11 acquires anothercamera image (step S11), it returns to the above-indicated step S7 so asto repeat execution of step S7 and subsequent steps. At this point,explanation of the detecting process according to the first embodimentof the invention is finished.

As described above, in the Botts-dots detecting process of thisembodiment, the size of the detection areas used in the initial findingstage and that of the detection areas used in the tracking stage arevaried or made different from each other. Namely, the detection areasare limited to a close region of the vehicle in the initial findingstage, for an improvement in the accuracy with which Botts Dots arefound, while controlling the processing load within a range that doesnot exceed the processing power of the ECU even if all-line scanning isperformed. Once Botts dots are found and their positions are grasped,the width of the detection areas is further reduced with reference tothe positions of the Botts dots, and a search for Botts dots is alsoconducted on a remote region as well as the close region of the vehicle.It is thus possible to increase the accuracy in detection of Botts dotsduring the tracking process while avoiding an increase in the overallprocessing load on the ECU. Thus, with regard to the Botts-dotsdetecting process that generally requires a large processing load forimage processing, it is possible to achieve highly accurate detection ofBotts dots while avoiding an increase in the overall processing load. Itis also possible to detect Botts dots with high accuracy even with theECU having relatively low processing power. Since highly accuratedetection of Botts dots is possible even if a high-powered ECU is notemployed, the cost of the vehicle may be advantageously reduced.

Next, a detection system according to a second embodiment of theinvention will be described. The functional block diagram of thedetection system is similar to that of the first embodiment (see FIG.1). In this embodiment, however, the NVRAM 12 stores a program that ispartially different from that of the first embodiment, and the ECU 11executes the program so as to implement a different function from thatof the first embodiment. More specifically, detection areas used in aprocess for detecting white lines are set in the same manner as that ofthe first embodiment for setting detection areas in the Botts-dotsdetecting process.

FIG. 15 is a flowchart showing details of a detecting process accordingto the second embodiment. In FIG. 15, the ECU 11 initially acquires acamera image (step S41), and sets initial detection areas 101 on thecamera image (step S42). More specifically, the ECU 11 reads initialdetection area data 134 from the RAM 13, and sets the initial detectionareas 101 on the camera image, based on the data.

Then, the ECU 11 performs a white-line detecting process with respect tothe initial detection areas 101 (step S43). This process issubstantially the same as that of step S3 as described above in thefirst embodiment; therefore, the process will not be described indetail.

Next, the ECU 11 determines whether any white line was detected within apredetermined time in step S43 (step S44). If it is determined that awhite line (or lines) was detected in the predetermined time in step S43(i.e., if an affirmative decision (YES) is obtained in step S44), theECU 11 sets “WHITE LINE” as the detection mode 138 (step S45). Then, theECU 11 proceeds to step S49 which will be described later.

If no white line was detected within the predetermined time in step S43(i.e., if a negative decision (NO) is obtained in step S44), the ECU 11performs a Botts-dots detecting process with respect to the initialdetection areas 101 (step S46). The process for detecting Botts dots issubstantially the same as that of step S23 as described above in thefirst embodiment; therefore, detailed description of the process willnot be provided herein.

Next, the ECU 11 determines whether Botts dots were detected within apredetermined time in step S46 (step S47). If it is determined thatBotts dots were detected within the predetermined time in step S46(i.e., if an affirmative decision (YES) is obtained in step S47), theECU 11 sets “BOTTS-DOTS” as the detection mode 138 (step S48). If, onthe other hand, no Botts dots were detected within the predeterminedtime in step S46 (i.e., if a negative decision (NO) is obtained in stepS47), the ECU 11 returns to the above-indicated step S41, and theroutine is repeated.

Next, the ECU 11 performs an operation to set the close detection areas102 (see FIG. 6) as described above (step S49). Furthermore, the ECU 11performs an operation to set the remote detection areas 103 (see FIG. 6)as described above (step S50). The operations of steps S49 and S50 aresubstantially the same as those of steps S26 and S27 of the firstembodiment, and therefore will not be explained herein.

Next, the ECU 11 determines whether the detection mode is “WHITE LINE”(step S51). If it is determined that the detection mode is “WHITE LINE”(i.e., if an affirmative decision (YES) is obtained in step S51), theECU 11 executes a white-line detecting process similar to that of stepS43, with respect to the close detection areas and remote detectionareas set in steps S49 and S50 (step S52). If, on the other hand, thedetection mode is not “WHITE LINE”, namely, if the detection mode is“BOTTS DOTS” (a negative decision (NO) is obtained in step S51), the ECU11 executes a Botts-dots detecting process similar to that of step S46,with respect to the close detection areas and remote detection areas(step S53).

If the process of step S52 or S53 is finished, the ECU 11 then performsa certain operation or function based on the detected road laneboundaries (white lines or Botts dots) (step S54). Thereafter, the ECU11 acquires another camera image (step S55), and returns to step S49 torepeat execution of step S49 and subsequent steps. At this point,explanation of the detecting process according to the second embodimentis finished.

As described above, in the second embodiment, the size of the detectionareas used in the initial finding stage and that of the detection areasused in the tracking stage are varied or made different from each other,during the white-line detecting process as well as the Botts-dotsdetecting process. As a result, the processing load required fordetection of white lines can be reduced. Consequently, the processingload required for processing associated with road lane boundaries can befurther reduced.

The road lane boundary detection system and detecting method accordingto the embodiments of the invention make it possible to detect road laneboundaries with high accuracy while avoiding an increase in theprocessing load, and are useful in, for example, a lane keeping assistsystem.

1. A road lane boundary detection system for detecting a road laneboundary provided on a road, from a road image that is a captured imageof a road ahead of a vehicle, comprising: a detection region settingunit that sets a certain region in the road image, as a target detectionregion to be searched for detection of the road lane boundary; and adetecting unit that processes image data in the target detection regionset by the detection region setting unit and that detects the road laneboundary, wherein the detection region setting unit includes: an initialdetection region setting unit that sets, in the road image, an initialdetection region of a certain size which is positioned close to thevehicle in real space and has a length in a width direction of the road,which is equal to or longer than a predetermined value, as the targetdetection region; a close detection region setting unit that sets aclose detection region that is positioned close to the vehicle in realspace, and has a length in the road width direction, which is shorterthan that of the initial detection region, as the target detectionregion; and a remote detection region setting unit that sets a remotedetection region that is positioned more remote from the vehicle thanthe close detection region in real space, and has a length in the roadwidth direction, which is shorter than that of the close detectionregion, as the target detection region. wherein the detecting unitprocesses image data in the initial detection region if no road laneboundary is detected, so as to detect a road lane boundary, andprocesses image data in the close detection region and the remotedetection region if the road lane boundary is detected, so as to detectthe road lane boundary, and the close detection region and the remotedetection region have the same size in the real space.
 2. The road laneboundary detection system according to claim 1, wherein: the closedetection region setting unit sets a position of the close detectionregion in the road width direction in the road image, based on aposition of the road lane boundary detected by the detecting unit usingthe initial detection region; and the remote detection region settingunit sets a position of the remote detection region in the road widthdirection in the road image, based on the position of the road laneboundary detected by the detecting unit using the initial detectionregion.
 3. The road lane boundary detection system according to claim 2,wherein: the close detection region setting unit sets the position ofthe close detection region in the road image, such that the road laneboundary detected by the detecting unit using the initial detectionregion is located at a center of the close detection region in the roadwidth direction; and the remote detection region setting unit sets theposition of the remote detection region in the road image, such that theroad lane boundary detected by the detecting unit using the initialdetection region is located at a center of the remote detection regionin the road width direction.
 4. The road lane boundary detection systemaccording to claim 1, wherein the target detection region comprises apair of regions that are separated from each other in the road widthdirection, and each of the regions has a width that contains acorresponding to one of left and right lane boundaries located to theleft and right of the vehicle.
 5. The road lane boundary detectionsystem according to claim 1, wherein the road lane boundary comprisesBotts dots.
 6. A road lane boundary detecting method for detecting aroad lane boundary provided on a road, from a road image that is acaptured image of a road ahead of a vehicle, comprising: a detectionregion setting process of setting a certain region in the road image, asa target detection region to be searched for detection of the road laneboundary; and a detecting process of processing image data in the targetdetection region and detecting the road lane boundary, wherein, when thedetection region setting process includes: an initial detection regionsetting process of setting, in the road image, an initial detectionregion of a certain size which is positioned close to the vehicle inreal space and has a length in a width direction of the road, which isequal to or longer than a predetermined value, as the target detectionregion; a close detection region setting process of setting a closedetection region that is positioned close to the vehicle in real space,and has a length in the road width direction, which is shorter than thatof the initial detection region, as the target detection region; and aremote detection region setting process of setting a remote detectionregion that is positioned more remote from the vehicle than the closedetection region in real space, and has a length in the road widthdirection, which is shorter than that of the close detection region, asthe target detection region, wherein, if no road lane boundary isdetected, image data in the initial detection region is processed so asto detect a road lane boundary, and if the road lane boundary isdetected, image data in the close detection region and the remotedetection region are processed so as to detect the road lane boundary,and the close detection region and the remote detection region have thesame size in the real space.